Shared Arrangements: practical inter-query sharing for streaming dataflows

Current systems for data-parallel, incremental processing and view maintenance over high-rate streams isolate the execution of independent queries. This creates unwanted redundancy and overhead in the presence of concurrent incrementally maintained queries: each query must independently maintain the same indexed state over the same input streams, and new queries must build this state from scratch before they can begin to emit their first results. This paper introduces shared arrangements: indexed views of maintained state that allow concurrent queries to reuse the same in-memory state without compromising data-parallel performance and scaling. We implement shared arrangements in a modern stream processor and show order-of-magnitude improvements in query response time and resource consumption for interactive queries against high-throughput streams, while also significantly improving performance in other domains including business analytics, graph processing, and program analysis

Autonomous-System Interfaces

The current Internet is characterized by a growing tension between the "core" (the Internet service providers) and the "edge" (the operators of edge networks and distributed applications). Much of this tension concerns path visibility and control -- where traffic goes (route control), where traffic comes from (path identification and filtering), and what happened in between (monitoring and accountability). We argue that this conflict harms both the core and the edge and that, to resolve it, we have to expose the Autonomous System (AS) as a first-class Internet object. This would map the functional structure of the Internet (the granularity at which edge systems can observe and control their traffic) to the organizational one (a graph of ASes). We argue that providing a well-defined interface between core and edge ASes offers significant benefits to both of them

Public Health for the Internet φ Towards A New Grand Challenge for Information Management

Business incentives have brought us within a small factor of achieving the database community\u27s Grand Challenge set out in the Asilomar Report of 1998. This paper makes the case for a new, focused Grand Challenge: Public Health for the Internet. The goal of PHI (or φ) is to enable collectives of hosts on the Internet to jointly monitor and promote network health by sharing information on network conditions in a peer-to-peer fashion. We argue that this will be a positive effort for the research community for a variety of reasons, both in terms of its technical reach and its societal impact. This version of the φ vision is targeted at readers in the database research community, but the effort is clearly multidisciplinary. A more generalist version of this paper will be maintained at http://openphi.net

Querying at Internet Scale

We are developing a distributed query processor called PIER, which is designed to run on the scale of the entire Internet. PIER utilizes a Distributed Hash Table (DHT) as its communication substrate in order to achieve scalability, reliability, decentralized control, and load balancing. PIER enhances DHTs with declarative and algebraic query interfaces, and underneath those interfaces implements multihop, in-network versions of joins, aggregation, recursion, and query/result dissemination. PIER is currently being used for diverse applications, including network monitoring, keyword-based filesharing search, and network topology mapping. We will demonstrate PIER\u27s functionality by showing system monitoring queries running on PlanetLab, a testbed of over 300 machines distributed across the globe